1. Field of the Invention
The present invention relates to a calibration technique of orientation sensors mounted on an image capturing apparatus and physical object.
2. Description of the Related Art
Studies about mixed reality that superimposes text and CG pictures on a physical object and that presents a result of superimposing have been extensively made. A presentation apparatus that presents mixed reality can be implemented as an apparatus which superposes and renders, onto a captured image of a physical object captured by an image capturing apparatus such as a video camera or the like, an image generated according to the relationship between the positions and orientations of the image capturing apparatus and physical object. In order to implement such presentation apparatus, the position and orientation of the physical object with respect to the image capturing apparatus need to be measured in real time.
It is a common practice to measure the position and orientation of the physical object with respect to the image capturing apparatus using an image captured by the image capturing apparatus. With this method, features such as markers and the like allocated on the physical object are detected on the captured image, and the position and orientation of the physical object are calculated based on the detected features.
A method which mounts orientation sensors on the image capturing apparatus and physical object, and uses the orientation measurement values by the orientation sensors and the captured image of the image capturing apparatus together has been proposed (see non-patent reference 1). With this method, the position and orientation of the physical object can be measured more stable than a case using the captured image alone.
The method disclosed in non-patent reference 1 is premised on that the orientations of the image capturing apparatus and physical object can be measured by the orientation sensors. However, the orientation measurement value actually output from each orientation sensor represents the orientation of the orientation sensor itself, but it is not the orientation itself of an object to be measured (image capturing apparatus and physical object). That is, the orientation measurement value of the orientation sensor cannot be used intact as the orientation of the object to be measured, and need to undergo some coordinate transformation. More specifically, the coordinate transformation that transforms the orientation of the orientation sensor itself into that of the object to be measured is required. This coordinate transformation can be defined by the orientation of the orientation sensor on a coordinate system with reference to the object to be measured (to be referred to as a measurement target object coordinate system hereinafter). The orientation of the orientation sensor on the measurement target object coordinate system will be referred to as “allocation information” of the orientation sensor hereinafter as needed. This allocation information will also be referred to as the “Local Transform” of the orientation sensor hereinafter. The operation and process for acquiring the allocation information of the orientation sensor will be referred to as “calibration” of the orientation sensor hereinafter.
One conventional calibration method is a trial-and-error method in which the operator interactively increases or decreases parameters that represent the Local Transform.
Patent reference 1 discloses a method of automatically calculating the allocation information of the orientation sensor mounted on the image capturing apparatus using images obtained by capturing markers allocated on a physical space from a plurality of viewpoints. With this method, the orientation sensor mounted on the image capturing apparatus can be calibrated by a very simple operation.
[Non-patent Reference 1] Kotake, Satoh, Uchiyama, and Yamamoto: “Hybrid Registration Using Inclination Constraint”, Papers of Meeting on Image Recognition and Understanding (MIRU2006), pp. 241-248, July 2006.
[Non-patent Reference 2] K. Satoh, S. Uchiyama, H. Yamamoto, and H. Tamura: “Robust vision-based registration utilizing bird's-eye view with user's view,” Proc. 2nd IEEE/ACM Int'l Symp. on Mixed and Augmented Reality (ISMAR 2003), pp. 46-55, October 2003.
[Non-patent Reference 3] A. I. Comport, E. Marchand, and F. Chaumette, “A real-time tracker for markerless augmented reality,” Proc. 2nd IEEE/ACM Int'l Symp. on Mixed and Augmented Reality (ISMAR 2003), pp. 36-45, October 2003.
[Patent Reference 1] Japanese Patent Laid-Open No. 2005-326275
The method disclosed in patent reference 1 cannot calibrate an orientation sensor mounted on an arbitrary physical object other than the image capturing apparatus (in other words, a physical object having no image capturing apparatus). Therefore, in the conventional system, when the orientation sensor is mounted on an arbitrary physical object other than the image capturing apparatus, the operator can adjust the Local Transform of this orientation sensor by only repeating trial and error processes. However, this operation is not easy, and requires advanced skills of the operator.
The method disclosed in patent reference 1 suffers a problem that parameters used as initial values must be estimated and input by an arbitrary method.
Also, the method disclosed in patent reference 1 suffers a problem that the gravity axis direction must be input in advance on a world coordinate system that defines the three-dimensional positions of markers allocated on the physical space.